<html>
<head><meta charset="utf-8"><title>meeting 2021-05-10 · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html">meeting 2021-05-10</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="238175878"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238175878" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238175878">(May 10 2021 at 16:57)</a>:</h4>
<p>.</p>



<a name="238176045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176045">(May 10 2021 at 16:58)</a>:</h4>
<p>o/</p>



<a name="238176269"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176269" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176269">(May 10 2021 at 17:00)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span> meeting time!</p>



<a name="238176299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176299">(May 10 2021 at 17:00)</a>:</h4>
<p>o/</p>



<a name="238176414"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176414" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176414">(May 10 2021 at 17:01)</a>:</h4>
<p>o/</p>



<a name="238176513"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176513" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176513">(May 10 2021 at 17:01)</a>:</h4>
<p>ok let's start!</p>



<a name="238176613"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176613" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176613">(May 10 2021 at 17:02)</a>:</h4>
<p>we have no issues to look at</p>



<a name="238176682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176682">(May 10 2021 at 17:03)</a>:</h4>
<p>I have a "discussion" item on the <a href="http://crates.io">crates.io</a> monitoring topic, does anyone else have other topics before dedicating potentially the rest of the meeting to <a href="http://crates.io">crates.io</a> monitoring?</p>



<a name="238176784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176784">(May 10 2021 at 17:03)</a>:</h4>
<p>Not me.</p>



<a name="238176932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238176932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238176932">(May 10 2021 at 17:04)</a>:</h4>
<p>so, I spent with my <a href="http://crates.io">crates.io</a> team hat last week probably too much time looking at the production database, and that pointed out a shortcoming on the plan we decided to go with regarding <a href="http://crates.io">crates.io</a> monitoring</p>



<a name="238177051"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177051" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177051">(May 10 2021 at 17:05)</a>:</h4>
<p>to be clear I did not have time yet to upgrade prometheus+grafana nor to implement the infra-side changes for <a href="http://crates.io">crates.io</a> monitoring</p>



<a name="238177147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177147">(May 10 2021 at 17:06)</a>:</h4>
<p>while investigating the performance issues we had with <a href="http://crates.io">crates.io</a> I needed to look at the Heroku Postgres metrics, and those are... annoying</p>



<a name="238177231"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177231" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177231">(May 10 2021 at 17:07)</a>:</h4>
<p>documentation on the postgresql metrics is <a href="https://devcenter.heroku.com/articles/heroku-postgres-metrics-logs">available here</a>, but the gist is that the only way to get the metrics is to scrape the logs and extract the metrics from lines like this one:</p>
<div class="codehilite"><pre><span></span><code>Nov 09 23:56:02 source=HEROKU_POSTGRESQL_SILVER addon=postgresql-devcenter-123456 sample#current_transaction=54017686 sample#db_size=16012956319bytes sample#tables=97 sample#active-connections=7 sample#waiting-connections=0 sample#index-cache-hit-rate=0.9239 sample#table-cache-hit-rate=0.93609 sample#load-avg-1m=0 sample#load-avg-5m=0 sample#load-avg-15m=0 sample#read-iops=0 sample#write-iops=0.13333 sample#tmp-disk-used=33849344 sample#tmp-disk-available=72944943104 sample#memory-total=4044960kB sample#memory-free=46920kB sample#memory-cached=3667532kB sample#memory-postgres=20824kB sample#wal-percentage-used=0.06512959334021144
</code></pre></div>



<a name="238177329"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177329" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177329">(May 10 2021 at 17:07)</a>:</h4>
<p>unfortunately those metrics are <em>only</em> available through logs, and heroku only graphs a couple of them in the dashboard, with no way I can see from the dashboard to get graphs from heroku</p>



<a name="238177462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177462">(May 10 2021 at 17:08)</a>:</h4>
<p>last week I wrote a quick script that fetches all of those lines from papertrail and generates an HTML table for me to look at, but we should really have those metrics in grafana</p>



<a name="238177578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177578">(May 10 2021 at 17:09)</a>:</h4>
<p>I didn't have time yet to see if there are off the shelf prometheus exporters for heroku postgresql, but I'm not hopeful there will be something suitable for us</p>



<a name="238177579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177579">(May 10 2021 at 17:09)</a>:</h4>
<p>is there a way to achieve that easily?</p>



<a name="238177594"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177594" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177594">(May 10 2021 at 17:09)</a>:</h4>
<p>ah</p>



<a name="238177684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177684">(May 10 2021 at 17:10)</a>:</h4>
<p>I guess if we're going to export from logs that brings the question of whether we should just pipe everything through that</p>



<a name="238177708"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177708" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177708">(May 10 2021 at 17:10)</a>:</h4>
<p>yep that was what I was going to bring up</p>



<a name="238177777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177777">(May 10 2021 at 17:10)</a>:</h4>
<p>I wonder if something like Loki or so would do it, though I know nothing about this space</p>



<a name="238177858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177858">(May 10 2021 at 17:11)</a>:</h4>
<p>(my understanding is that loki would be a replacement for papertrail)</p>



<a name="238177939"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238177939" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238177939">(May 10 2021 at 17:11)</a>:</h4>
<p>I have been lightly using loki recently, and I believe it requires jsonline logs</p>



<a name="238178005"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178005" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178005">(May 10 2021 at 17:12)</a>:</h4>
<p>if there wasn't heroku postgresql to monitor I'd still be convinced installing prometheus on dynos would've been the best approach</p>



<a name="238178026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178026">(May 10 2021 at 17:12)</a>:</h4>
<p>Q: can we query postgres ourselves and expose these as service level metrics?</p>



<a name="238178046"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178046" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178046">(May 10 2021 at 17:12)</a>:</h4>
<p>that'd be too easy</p>



<a name="238178067"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178067" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178067">(May 10 2021 at 17:12)</a>:</h4>
<p>(since the prior solution was simpler than log analysis)</p>



<a name="238178101"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178101" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178101">(May 10 2021 at 17:12)</a>:</h4>
<p>as far as I'm aware we could do that for some metrics, probably all the ones related to the database</p>



<a name="238178138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178138">(May 10 2021 at 17:12)</a>:</h4>
<p>but it wouldn't expose the metrics I care the most about right now, which are load average and iops</p>



<a name="238178256"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178256" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178256">(May 10 2021 at 17:13)</a>:</h4>
<p>Hm.</p>



<a name="238178363"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178363" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178363">(May 10 2021 at 17:14)</a>:</h4>
<p>Well, I guess log analysis is the path then</p>



<a name="238178462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178462">(May 10 2021 at 17:15)</a>:</h4>
<p>Though I admit to feeling like something is wrong - does everyone handroll this?</p>



<a name="238178572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178572">(May 10 2021 at 17:15)</a>:</h4>
<p>I'll surely do more research on this before starting to write any line of code, I don't want to go down this route unless we absolutely need to either</p>



<a name="238178574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178574">(May 10 2021 at 17:15)</a>:</h4>
<p>sorry, I think I lost the thread of conversation:</p>
<ol>
<li>pietro was originally suggesting prom on dyno</li>
<li>that doesn't work because the postgres plugin exposes things via logs</li>
<li>simulacrum suggests putting postgres info from the <a href="http://crates.io">crates.io</a> service</li>
<li>pietro says we could do that but it wouldn't work for load avg+iops</li>
</ol>
<p>couldn't we do 4, with 1 as well to compensate for load avg and iops?</p>



<a name="238178702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178702">(May 10 2021 at 17:16)</a>:</h4>
<p>(I may have misunderstood what service level metrics referred to)</p>



<a name="238178730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178730">(May 10 2021 at 17:16)</a>:</h4>
<p>Postgres isn't a dyno under our control so I think (1) doesn't work for it</p>



<a name="238178776"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178776" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178776">(May 10 2021 at 17:16)</a>:</h4>
<p>ooooo, load avg of the <em>postgres</em> dyno. gotcha</p>



<a name="238178792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178792">(May 10 2021 at 17:17)</a>:</h4>
<p>should've been more clear :)</p>



<a name="238178824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178824">(May 10 2021 at 17:17)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-10/near/238177231">said</a>:</p>
<blockquote>
<p>documentation on the postgresql metrics is <a href="https://devcenter.heroku.com/articles/heroku-postgres-metrics-logs">available here</a>, but the gist is that the only way to get the metrics is to scrape the logs and extract the metrics from lines like this one:</p>
<p><div class="codehilite"><pre><span></span><code>Nov 09 23:56:02 source=HEROKU_POSTGRESQL_SILVER addon=postgresql-devcenter-123456 sample#current_transaction=54017686 sample#db_size=16012956319bytes sample#tables=97 sample#active-connections=7 sample#waiting-connections=0 sample#index-cache-hit-rate=0.9239 sample#table-cache-hit-rate=0.93609 sample#load-avg-1m=0 sample#load-avg-5m=0 sample#load-avg-15m=0 sample#read-iops=0 sample#write-iops=0.13333 sample#tmp-disk-used=33849344 sample#tmp-disk-available=72944943104 sample#memory-total=4044960kB sample#memory-free=46920kB sample#memory-cached=3667532kB sample#memory-postgres=20824kB sample#wal-percentage-used=0.06512959334021144
</code></pre></div><br>
</p>
</blockquote>
<p>Unwrapping from code blocks for easier reading: Nov 09 23:56:02 source=HEROKU_POSTGRESQL_SILVER addon=postgresql-devcenter-123456 sample#current_transaction=54017686 sample#db_size=16012956319bytes sample#tables=97 sample#active-connections=7 sample#waiting-connections=0 sample#index-cache-hit-rate=0.9239 sample#table-cache-hit-rate=0.93609 sample#load-avg-1m=0 sample#load-avg-5m=0 sample#load-avg-15m=0 sample#read-iops=0 sample#write-iops=0.13333 sample#tmp-disk-used=33849344 sample#tmp-disk-available=72944943104 sample#memory-total=4044960kB sample#memory-free=46920kB sample#memory-cached=3667532kB sample#memory-postgres=20824kB sample#wal-percentage-used=0.06512959334021144</p>



<a name="238178931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178931">(May 10 2021 at 17:17)</a>:</h4>
<p>yeah ok that's rough</p>



<a name="238178988"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238178988" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238178988">(May 10 2021 at 17:18)</a>:</h4>
<p><a href="https://elements.heroku.com/addons/librato">https://elements.heroku.com/addons/librato</a> claims to do everything for us, although in a different platform</p>



<a name="238179015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179015">(May 10 2021 at 17:18)</a>:</h4>
<p>(just from a quick look)</p>



<a name="238179297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179297">(May 10 2021 at 17:20)</a>:</h4>
<p>my bias is (always) to tend towards getting a lop processing pipeline in place because it becomes a hammer you can use for tons of stuff</p>



<a name="238179401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179401">(May 10 2021 at 17:20)</a>:</h4>
<p>the really annoying thing is that we have these things we need to monitor:</p>
<ol>
<li>heroku postgres, can only do that via logs</li>
<li>instance-level metrics, can do it either way</li>
<li>node_exporter on our dynos, can only do it via prometheus on dynos</li>
</ol>



<a name="238179412"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179412" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179412">(May 10 2021 at 17:20)</a>:</h4>
<p>but...I'm not actually sure what that would look like. sounds like it might require infra off heroku</p>



<a name="238179444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179444">(May 10 2021 at 17:21)</a>:</h4>
<p>I would definitely deploy it on our AWS ECS cluster</p>



<a name="238179507"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179507" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179507">(May 10 2021 at 17:21)</a>:</h4>
<p>and configure heroku to POST logs there (it's actually really easy to do so)</p>



<a name="238179583"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179583" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179583">(May 10 2021 at 17:21)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-10/near/238179401">said</a>:</p>
<blockquote>
<p>the really annoying thing is that we have these things we need to monitor:</p>
<ol>
<li>heroku postgres, can only do that via logs</li>
<li>instance-level metrics, can do it either way</li>
<li>node_exporter on our dynos, can only do it via prometheus on dynos</li>
</ol>
</blockquote>
<p>(3) is not critical, and if we need to montitor some specific system stats we can reimplement those parts of node_exporter on <a href="http://crates.io">crates.io</a> I guess</p>



<a name="238179702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179702">(May 10 2021 at 17:22)</a>:</h4>
<p>Yeah log processing is definitely something I think we'll eventually want, for lots of reasons</p>



<a name="238179730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179730">(May 10 2021 at 17:22)</a>:</h4>
<p>But it sounds like it's not a complete solution anywhere</p>



<a name="238179839"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179839" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179839">(May 10 2021 at 17:23)</a>:</h4>
<p>can you elaborate on the difference between 2 and 3? I thought node_exporter was for instance-level metrics</p>



<a name="238179851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179851">(May 10 2021 at 17:23)</a>:</h4>
<p>sure</p>



<a name="238179905"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179905" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179905">(May 10 2021 at 17:24)</a>:</h4>
<p>we have crates.io-specific instance metrics, like what's the status of the connection pool or how many requests we're processing, and those are implemented in the app</p>



<a name="238179980"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238179980" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238179980">(May 10 2021 at 17:24)</a>:</h4>
<p>then it'd be nice to have node_exporter stats to know how the underlying dyno is going</p>



<a name="238180062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180062">(May 10 2021 at 17:25)</a>:</h4>
<p>right gotcha</p>



<a name="238180240"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180240" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180240">(May 10 2021 at 17:26)</a>:</h4>
<p>I guess, thankfully we did have <a href="http://crates.io">crates.io</a> outages before we started installing prometheus on dynos</p>



<a name="238180354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180354">(May 10 2021 at 17:27)</a>:</h4>
<p>ok then, seems we have little choice</p>



<a name="238180381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180381">(May 10 2021 at 17:27)</a>:</h4>
<p>nothing like reality to break plans <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="238180404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180404">(May 10 2021 at 17:27)</a>:</h4>
<p>I'll rely this to the <a href="http://crates.io">crates.io</a> team and then the <a href="http://crates.io">crates.io</a> team will pick up on the impl work</p>



<a name="238180432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180432">(May 10 2021 at 17:27)</a>:</h4>
<p>we'll probably deploy this to the ECS cluster</p>



<a name="238180434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180434">(May 10 2021 at 17:27)</a>:</h4>
<p>hm just to ask - what exactly are we implementing?</p>



<a name="238180466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180466">(May 10 2021 at 17:27)</a>:</h4>
<p>let me rephrase that</p>



<a name="238180469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180469">(May 10 2021 at 17:27)</a>:</h4>
<p>It sounded to me like the next step <em>should</em> be "is there any off the shelf solution"</p>



<a name="238180522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180522">(May 10 2021 at 17:28)</a>:</h4>
<p>yeah forgot that bit <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="238180593"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180593" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180593">(May 10 2021 at 17:28)</a>:</h4>
<p>if there is no off the shelf solution we'll probably leave impl to the <a href="http://crates.io">crates.io</a> team and deploy it on AWS</p>



<a name="238180601"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180601" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180601">(May 10 2021 at 17:28)</a>:</h4>
<p>and it may even be worth, like, an internals post - I think it's pretty clear most of us here are not really experts on this stuff, and maybe someone will tell us about thing Y</p>



<a name="238180655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180655">(May 10 2021 at 17:28)</a>:</h4>
<p>and/or a question for support</p>



<a name="238180773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180773">(May 10 2021 at 17:29)</a>:</h4>
<p>pity heroku don't have forums</p>



<a name="238180946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180946">(May 10 2021 at 17:30)</a>:</h4>
<p>ok then, that's a plan</p>



<a name="238180960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180960">(May 10 2021 at 17:30)</a>:</h4>
<p>thank y'all for attending!</p>



<a name="238180961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238180961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238180961">(May 10 2021 at 17:30)</a>:</h4>
<p>\o</p>



<a name="238181001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181001">(May 10 2021 at 17:31)</a>:</h4>
<p>internals could be worth a shot. I can compose something, it'll be a good test of my understanding</p>



<a name="238181027"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181027" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181027">(May 10 2021 at 17:31)</a>:</h4>
<p>\o</p>



<a name="238181042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181042">(May 10 2021 at 17:31)</a>:</h4>
<p>\o</p>



<a name="238181120"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181120" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181120">(May 10 2021 at 17:31)</a>:</h4>
<p>\o</p>



<a name="238181184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181184">(May 10 2021 at 17:32)</a>:</h4>
<p><span class="user-mention silent" data-user-id="117568">Aidan Hobson Sayers</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-10/near/238181001">said</a>:</p>
<blockquote>
<p>internals could be worth a shot. I can compose something, it'll be a good test of my understanding</p>
</blockquote>
<p><span aria-label="thumbs up" class="emoji emoji-1f44d" role="img" title="thumbs up">:thumbs_up:</span> post it here before internals for double-checking, thanks <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span></p>



<a name="238181259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181259">(May 10 2021 at 17:32)</a>:</h4>
<p>ah yep, should've mentioned that part of the plan!</p>



<a name="238181736"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-10/near/238181736" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-10.html#238181736">(May 10 2021 at 17:35)</a>:</h4>
<p>\o</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>